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Several yaars ago Durzo, Diamond, and Doughty (1979) provided a lucid 
analysis of research needs in instructional development. This paper aims to 
reassess and update the status of research on the broad process of instruc-- 
tional develonment and to suggest an agenda for future inquiry. It will 
analyze several conceptual problems that have constrained previous research, 
suggest resolutions of these problems, and provide a detailed framework for 
future research, including specific questions and suggestions for methods of 
inquiry. 



Definitional Problem 

Much to the f*<itriment of communication among scholars, the terms "instruc- 
tional design" and "instructional development" have been used incerchangeably 
by many researchers and theoreticians over the past fifteen years- This prob- 
lem has persisted even though the earliest papers in this field originally 
used "instructional development" as the broader term (e.g., Paris, 1968) and 
despite the fact that the most widely accepted definitional schena in the 
field of educational technology also defines instructional design as a subset 
of instructional development (AECT, 1977), Further, some authors fail to 
distinguish instructional development from the related processes of faculty 
development, student development, organizational development, and the like. 

Two recent works have contributed greatly to a renewed clarification of 
this cluster of related concepts. First, Reigeluth's (1983) highly regarded 
(and aptly titled) book, Instructional-Design Theories and Models does indeed 
focus on instructional design ii:sues. In it Reigeluth takes pains to distin- 
guish instructional design from instructional development. When used technic- 
ally, these terms are defined as follows: 

Instructional design ... is the process of deciding what methods of 
instruction are best for bringing about desired changes in student 
knowledge and skills for a specific course content and a specific student 
population (p. 7). 

Instructional development Is the process of prescribing and using optimal 
procedures for creating naw instruction in a given situation (p. 8). 

When speaking generally of the whole process of which these two are compo- 
nents, Reigeluth uses "instructional development" as che superordinate term. 

Second, Bass and Dills (1984) entitle their wide-ranging anthology as 
Instructional Development; The State of the Art . The collection includes 
Robert Braden's seminal paper, "A Place in Space: ID's Universe," which has 
been hailed by reviewers for its definitional clarity. In it, the broad 
concept of instructional development is distinguished from the related 
concepts of faculty development (modifying attitudes and skills of faculty), 
organizational development (organization-wide effort to implement controlled 
change), student development (providing student with additional learning 
tools), and context development (planned efforts to change the environment 
surrounding an instructional system). 

The importance of defining terms clearly is illustrated by Hannafin's 
recent overview (1986) of current and future directions of research. He 
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variously refers to his focus of attention as "instructional design and 
technology," "instructional development and technology," "instructional 
developmeat "design of instruction,'* and "the ID field" (exact referent of 
the acronym is unspecified). His generalizations are often difficult to 
evaluate because the referent is unclear. For example: "Perhaps due to the 
strong influence of the behavioral sciences on the ID field . • • experimen- 
tal paradigms have dominated published research" (p, 25)* This may be true 
of instructional design, but not of instructional devslopment. In fact, 
Hannafin raises a number of legitimate issues and gives a useful set of recom- 
mendations for encouraging more and better research in any field. But the 
prescriptions are generic; this is expectable given the "shotgun" diffuseness 
of the diagnosis. This paper aims to suggest a more specific set of recommen- 
dations regarding a framework and agenda for future instructional development 
research. 

For the sake of clearer communication, I propose that the term "instruc- 
tional development" be used to refer to the systematic process of analyzing, 
designing, producing, evaluating, and implementing instructional systems or 
components thereof. By this definition, " instructional design" is a subset 
of instructional development, referring to that part of che process devoted 
to specifying instructional treatments most appropriate for attaining parti- 
cular objectives by a given set of learners. It is the part of the process 
that fits between learner/task analysis and selection/ production of methods/ 
materials. Its outcome is the set of specifications for the methods and 
materials. 



Review of Previous Research 

Two recent critiques of research in educational/instructional technology 
by Winn (1986) and Gerlach (1984) provide a convenient point of departure for 
a review of previous research on instructional development (ID). First, both 
Winn and Gerlach deal with the whole domain of eaucational/instructional tech- 
nology and they both implicitly treat the issues of "learning from media" as 
the Heartland of the field; however, they do allude to instructional design/ 
development as a major subset of the larger field. Unfortunately for our 
present purposes, they both use the terms desigi. and development interchange- 
ably. Inferring from the variables they discuss and the examples they use, 
it appears that both authors usually are focusing on what is here called 
instructional design. For example, in referring to recent findings on 
"instructional development," Winn says, "We are beginning co discover the 
relationships among task, student characteristics, methods and strategies 
that students actually use" (pp. 8-9). Such a statement indicates a fixation 
with design issues, as contrasted with the broad concerns—social, political, 
and economic — that are salient to a complex process conducted by people in 
various human settings. 

Because of such biases in recent research syntheses it is difficult lO 
get a sense of where we stand in instructional development research above and 
beyond the design issues. Consequently, I recently undertook an independent 
review of research with the assistance of several graduate students. Our 
search centered around the scholarly journals in the field of educational/ 
instructional technology plus relevant dissertations catalogued in Disserta- 
tion Abstracts International . One of the more promising bodies of literature 



Agenda for Research - 3 



that has not been adequately explored is that of government/military research 
on ID, Two very recent articles, by McCombs (1986) and Ellson (1986;, indi- 
cate that there is a substantial reservoir of govermnert-sponsored research 
on xD conducted in military and foreign educational settings that has not yet 
been integrated adequately into the academic mainstream, McCombs' s and 
Ellson* s observations are referred to later in this paper, but the primary 
sources have not yet been reviewed. 

In general, if one excludes studies dealing primarily with instructional 
design concerns (relationships among various treatments, learners, and 
tasks), the major scholarly journals closest to the instructional development 
(ID) field carry relatively few reports of research on ID> That is, few 
articles were found that reported quantitative or qualitative findings on the 
ID process itself; the sub-processes of analysis, selection/production, evalu- 
ation, or implementation; professional roles of instructional developers; or 
socioeconomic aspects of ID, Six such studies were fo--nd in Journal of 
Instructional Development since 1978 (Nalbone, 1979; Guzy et al . , 1979; Klein 
& Doughty, 1980; Vanek & Kennedy, 1981; Willis, 1983; Higg ins & Reiser, 
1985); of these, four are basically case studies, 

In Educational Communication and Technology Journal (and its predecessor 
AV Connnunication Review ) since 1974, six studies reported data on ID pheno- 
mena (Hoban, 1974; Savage, 1975; Kandaswamy et al . , 1975; Kerr, 1977; Burton 
& Aversa, 1979; Shrock, 1985); another (McCombs, 1986) synthesized evidence 
from earlier field research, 

^SPI's Improving Human Performance Quarterly in the 1970s and — to a les- 
ser extent— Performance & Instruction in the 1980s have also carried a number 
of reports based on empirical research on ID (Short, 1973; Smith et al . , 
1976; Nathenson et al • , 1977; Smith, 1978a; Smith 1978b; Krug et ai . , 1979; 
Golas, 1983; Wager, 1983), 

Other journals, including Educational Technology , Phi Delta Kappan , and 
Educational Psychologist > carry ID research articles on occasion. But, even 
looking only at the mainstream journals, ID research occupies a rather small 
niche, representing no more than 5 percent of the total articles carried in 
these journals. 

In sheer quantity, unpublished dissertations comprise the single largest 
source of research on ID, contributing perhaps half of the total "knowledge 
base" in this field. Eighteen dissertations are cited in the References, rep- 
resenting those studies that fell closest to the domain of ID as defined here 
(again, excluding instructional design studies) and which most clearly are 
oased on observational data gathered under laboratory or field conditions. 
Excluded are a dozen or more questionnaire surveys which seemed to serve no 
theoretical purpose beyond describing some audience's ''awareness'* or "atti- 
tude toward" ID, 'Examples of the dissertations included in this review are: 
Patterson (1981) who interviewed developers and examined documents in ten 
corporate settings to determine the extent to which they employed ID proce- 
dures; Scudder (1982) who used a questionnaire to survey corporate instruc- 
tional developers to determine the extent of their use of ID procedures; 
Orban (1982) who did an ethnographic study, observing the consultation inter- 
actions between a developer an'' several clients; and Holsclaw (1974) who 
surveyed thirteen ID agencies in higher education concerning their working 
guidelines, distilling these into over 100 "heuristics" for conducting ID, 
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It's difficult to generalize about the set of studies gathered in this 
:earch. They represent every point on the quantitative/qualitative spectrum, 
from Kandaswamy, Wager, and Nathenson on the experimental design end of the 
scale to Shrock and Chen on the naturalistic end, V/hat may be most notable 
is that there is no prevailing paradigm evident. Cortrary to what other 
reviewers consistently claim is the pattern in media and instructional design 
research, the experimental psychological paradigm is followed in a minority 
of these studies. It appears that there is still a ready market for exemp- 
lars to set the pattern(s) for future research in this area. The field does 
not appear to have yet decided what is a "good" model for instructional 
development research. 



\^ew Approaches Advocate d 

At least since Salomon and Clark's (1977) critique of the methodology of 
research on instructional media and technology there has been '.n awareness of 
the limitations inherent in experimental laboratory research in the whole do- 
main of Educational Technology, Salomon and Clark focused on the dilemma of 
internal vs. external validity. As they put it, "the more it [media re- 
search] moved into the daeper layers of understanding media, the farther it 
'^ent from the world of education" (p. 106). Their principal recommendation 
was to buttress external validity by -mving toward research in natural set- 
tings, and they suggested several pseudo-experimental designs to maintain 
some degree of inte-nal validity. 

Salomon and Clark's admonition has been echoed with variations by many 
voices since then. Driscoll (1984) compiled an extensive list of alternative 
research paradigms, each with example studies, suitable to different ques- 
tions in the domain of "instructional systems." She mentioned quasi- 
experimentation, meta-analysis, case study and ethnography, systems-based 
evaluation, cost-effectiveness analysis, and technique, s and model develop- 
ment. Although some of her examples could be construed as inOs.ructional 
development (ID) research, most fall into tht larger domain of Educational 
Technology or the instructional design subset. 

Heinich's (1984) classic essay on the proper concerns of instructional 
technology speaks to ID as well in calling for systems analyses and other 
such "engineering" type decision-oriented studies. His main appeal is that 
the very nature of the instructional technology field, being based in profes- 
sional practice, demands a shift away frcm conclusion-oriented research, 
which so far has contributed more to the knowledge base of educational psy- 
chology than to instructional technology. Incidentally, Cunningham (1986) in 
his recent critique of "method A versus method B" comparison studies, ques- 
tions the assumptions such studies are based on and denies that they have 
contributed to the knowledge base even of educational psychology. 

Stephen Kerr (1985), in his introduction to a theme issue of ECTJ points 
out a "blind spot" in the study of educational communications and technology: 
"questions relating to the ways in which educational technology affects the 
social relationships among those vho work and learn in educational institu- 
tions and hov it may change the nature of those institutions themselves" (p. 
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The importance of social issues has been pointed out most p "masively 
and most directly for instructional development by Schwen an^ his colleagues 
(1984) • They emphasize that the practice of ID demands an understanding of 
social processes since ID is itself a social process (i.e., team planning) 
and a social intervention (i.e., working with instructor clients) which 
invariably takes place within a complex social system. 

A Conceptual Framework for ID Research 

In their perceptive analysis of ID research needs, Durzo, Diamond, and 
Doughty (1979) defined their domain to include not only instructional design 
as a subset of instructional development, but also faculty development (FD) 
and organizational development (OD), This attempt to update their work will 
specifically exclude FD and OD simply because they are clearly separate con- 
structs (as Braden argues) with different critical attributes. I will also 
differ from Durzo, Diamond, and Doughty by restricting the concept of 
''instructional design** more narrowly than they, limiting it (as Reigeluth 
argues) to the specification of instructional treatments. Finally, I will 
follow t-he lead of Heinich^ Kerr, and Schwen et al . by expanding ID's social 
concen ^ beyond the project management and client relationship categories 
suggested by Durzo, Diamond, and Doughty. 

Taking into consideration the framework proposed by Durzo, Diamond, and 
Doughty, adding to it the topics covered in previous research, and the social 
issues raised by Kerr, HeinicV , and Schwen et al . , and filling in obvious 
gaps, I have developed the following gc-neral framework as a beginning point 
in the search for a comprel»ensive framework for future ID research. I will 
first present a broad outline, then flesh in the details, section by section. 
Note that the outline is divided into two parts: Decision-oriented issues 
and Conclusion-oriented issues. \ot only do these two directions appeal to 
different consumers, but they also imply qualitatively dirferent questions. 

DECISION-ORIENTED ISSUES 

T. Administrative and policy issues of ID agencies 

II. Internal organization/management of ID teams 

III. Interaction with clier s 

IV. Social/political relationiships with supra-systems 

V. Optimization of ID procedures 

A. Overall ID model 

B. Needs analysis 

C. Learner analysis 

D. Task analysis 
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E* Environmental analysis 
?• Ob jeccives specification 

G. Instructional design (NOT ELABORATED IN THIS PAPER) 

H. Prototype construction 

I. Formative evaluation/revision 
J» Sumnative evaluation/revision 

K. Implementation of developed instruction 

CONCLUSION-ORIENTED ISSUES 

I. Definition of insti uctional development (ID) 

II. The value of ID 

Let us proceed to flesh out the decails of these issues, their sub- 
issues, and the research questions that they imply. 

I. Administrative and policy issues of ID agencies , Durzo, Diamond, and 
Doughty suggest a number of issues to be pursued in this category, 

A. The costs and benefits of ID 

1. What organizations should consider doing ID? 

2. What are the costs of doing ID? 
1. What are the benefits of ID? 

A. How can cost/benefit tradeoffs be calculated? 

5. How much time does ID require? 

6. What variables in the ID process affect time expenditures? Money 
costs? 

B. Administrative setup 

1. Where should an ID agency be placed in the organizational 
hierarchy? 

2. Who should direct an ID agency? What rank in the organization? 
Repc^-ting to whom? 

3. What should be the organizational relationship between ID agency 
and production agency? evaluation agency? training agency? other 
related agencies? 



ERLC 



9 



Agenda for Research - 7 



Role of ID agency 

1. What services are valued by inscruccor-cliencs? 

2. What services are valued by adniniscracor-cliencs 

3. What mix of services is mosc likely to make the largest 
instructional improvement impact? 

4. What mix of services is most likely to lead to longevity of the 
agency? 

characteristics of ID agency 

1. What competencies should be represented on the staff? 

2. What variables affect the optimal staff size (e.g., .size of 
supra-system in terms of budget, employees; number of projects 
undertaken)? 

3. On what basis sho ild ID services be centralized or decentralized? 
Approaches taken to ID 

1. What are the tradeoffs involved in emphasizing product development 
vs. training clients to solve their own problems? What are the 
cost? of each approach? Benefits of each? 

2. What attitudes or skilli> should be included in client training? 
Project Generation/ Selection 

1. Is it more productive to pursue a few large projects or aany small 
ones? Which approach contributes more to overall instructional 
improvement? Which leads to greater longevity of the ID agency? 

2. Should projects be generated internally from an <'?^alysis of 
organizational needs or externally from "walk-in" requests by 
instructor-clients? 

3. What diffusion strategy generates more project proposals? 

4. What sorts of incentives attract clients? What fac^ ;rs in the 
organizational climate have an effect on who supports ID and how 
strongly? 

5. By what criteria should project proposals be evaluated end 
prioritized? 

6. Who should set criteria for prioritization? What are the 
advantages/disadvantages of advisory boards? 
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II* Internal organizat ion/manaf>ement: of ID teams . 

A. Personnel 

1. What competencies are needed on different types of project teams? 

2. How many people can work productively on one project? Does adding 
staff Increase or reduce the time needed -to complete a project? 

1. Who should lead the ID team? 

A- What group dynamics skills are necessary for team members? 

B. Budgeting: time and money 

1. What administrative factors affect the cost of ID? How? 

2. WhaC factors affect manpower needs? Vhat factors determine time 
requirements? 

€• Internal organization 

I. Under what sorts of supervision and operating rules do different 
teams work best? Democratic? Authoritarian? Laissez-faire? 

2- What are useful methods of team building? . . . decision making? 
• . . sharing power/control? . . . conflict management? 

3. What sorts of documentation should be carried on routinely? . . . 
')y teams? . . • by agency? 

Ill, Interaction with clients . 

A. Client attributes 

1. Who volunteer to be ID clients? 

2. What attributes of clients tend to be functional/dysfunctional? 

3. On w'la t basis should clients be selected? 

B. Develouer attributes 

1. What developer attributes tend to be functional/dysfunctional? 

2. How can rapport be established? To what extent and in what ways 
does rapport contribute to projecc success? 

C. Strategies of interaction 

1. Are there predictable phases in the developer-client psychological 
relationship? How can these be recognized? What strategies can be 
used to manajje this relationship? 
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2. What are the advantages/disadvantages of various strategies of 
managing this relationship (e.g», regarding goal-setting, decision 
making)? 

3. What are appropriate tactics for each strategy (i.e., components of 
"consulting style")? 

IV. Social/political relationships with snpra-sys tems . 

.1. Organizational settings 

1. How can the various organizational settings be classified (e.g., 
publ ic/ private school , public/pr iva te col lege , corporation, 
government agency)? 

2. What attributes of the setting have an effect on acceptance or 
productivity of ID activities (e.g, "traditional" vs. "innovative" 
culture ;) ? 

3. What elements in the organizational setting generate pol i tlcal 
consequences for ID? Which elements in the supra-sys tem? ... in 
the supra-supra-systeu? ... in higher echelons? 

A. What elements within or beyond the organization generate economic 
consequences for ID? Which are short-tern, which are long-term? 

B. Goals and values 

1- How do the goals/values of ID correspond with the goals/values of 
the different actors? 

2. How receptive .ire different "corporate cultures" to the values of 
ID? 

3. Do different actors (e.g., administrators vs. instructors) differ 
in their acceptance of the values of ID? 

A. Do unionized and non-unionized organizations differ in their 
receptivity to ID? 

C. Political strategies 

1. What strategies can be used to *^hance the social/political/ 
economic position of ID agencies (e.g., role of advisory boards)? 

2. What are the political consequences of different operating strate- 
gies (e.g., project selection policies)? 

3. What are the social/political/economic incenti^^es of the different 
actors? 
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T)* ID effects on the organization 

1. What effects does an ID agency have on the organization (e.g,, as a 
model of rational decision-making)? 

2. es having an effect on the organization create political problems 
(e.g«, professional jealousy)? 

K« National cultures 

1. To what extent do different national cultures affect the acceptance 
of ID? 

2. Do attributes of different national cultures affect the productivi- 
ty of ID agencies? 

V* Optimization of ID procedures . As is the case with any field of prac- 
tice, a good deal of the inquiry in ID has been conducted with the objective 
Ol improving professional practice. The concerns in this area revolve around 
identifying useful techniques for doing ID, 

Many of the early dissertations in the ID field were aimed at discovering 
and testing overall models of ID (Stowe, 1971; Belmore, 1973; Holsclaw, 
1974), Very recently, at least two knowledgeable researchers have felt that 
sufficient evidence exists to make a judgment as to the efficacy of the most 
prominent ID models, Douglas Ellson (1986) in his systematic review of re- 
search studies in which technological instructional methods have been com- 
pared with conventional methods identifies several broad techniques (e,g,, 
programmed learning and programmed teaching) that consistently yield higher 
"relative productivity ratios" than conventional instruction, i\mong such 
successful techniques he includes "performance-based instructional design" in 
which "information obtained in one tryout of a particular teaching procedure 
with one group of pupils is ucad as the basis for revising the design of that 
procedure for the group, , ," (p, 119), 

Ellson cites as the basis for his judgment reports of several large-scale 
applications of ID procedures in elementary schools in the U,S,, in elementa- 
ry schools in Southeast Asia, and in U,S, military training. Interestingly, 
the military ID model that he cites, "Interservice Procedures for Instruction- 
al Systems Development (ISD)" is the same one examined in depth by McCombs 
(1986), McCombs surveys a much broader range of literature on ISD and finds 
that there is a widespread perception, especially in the military, of the 
failure of the ISD model. She attributes this perception to the users' "fail- 
ure to maintain a total systems perspective and of reducing the problem focus 
to the development of self-paced or individualized materials" (p, 71), She 
feels that systematic ID must be a complex, creative process applying higher 
order analytical skills. Reduced to mere routine procedures (as is the temp- 
tation in bureaucratized ID agencies) the model loses its effectiveness, 
McCombs charts a course for continuing research on the overall effectiveness 
of ID models by generating a set of "empirically identified factors in the 
successful implementation of ID," 

Questions in this area can be clustered first under the general heading 
of TD models overall, and then under headings signifying each of the elements 
in the ID process. 
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A. ID models overall 

1. How can different ID models be classified? 

2. Do different models differ in their effects, given that each is 
applied conscien tiously? 

3. What factors are associated with greater impact on instructional 
improvement? . • . on continued support of ID activities? 

4. What are the relative strengths and limitations of each type of ID 
model? 

5. On what basis should a developer select a model to follow? 

B. ?s^eeds analysis: What are optimal procedures for needs analysis? 

C. Learner analysis: What are optimal procedures -'or learner analysis? 

D. Task analysis: What are optimal procedures for task analysis? 

£• Environmental analysis: \^at are optimal procedures for environmental 
analysis? 

F. Objectives specifications: What are optimal procedures for objectives 
specification? 

G. Instructional design: What are optimal procedures for instructional 
design? That is, what design rules yield better decisions about 
matching treatment with audience with learning task? 

H. Prototype construction: What are optimal procedures for prototype 
construction? 

I. Formative evaluation/ revision: What are optimal procedures for 
forma five evalua t ion/ revision? 

J. Summative evaluation: l^hat are optimal procedures for suramative 
evaluation? 

K. Implementation: What are optimal procedures for implementing 
developed instruction? 

CONCLUSION-ORIENTED ISSUES 

Researchers, especially those approaching ID from the perspective of edu- 
cational psychology, tend to begin by raising conclusion-oriented questions. 
Their agenda is dominated by a concern for reaching conclusions about the 
value of ID relative to other instructional planning methods. This perspec- 
tive is typified by studies comparing "method A" with "method B." Such stu- 
dies often make implicit assumptions about the hypothetical constructs of 
"instructional development" and "conventional instruction." They assume that 
these constructs actually exist and they have certain attributes. The 
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problens wich chese assumptions have been expressed cogently by Gerlach and 
by Clark in a number of articles over the past decade; Gerlach (1984) and 
Clark (1985) are recent presentations of these issues. 

It is at least equally logical to defer conclusion-oriented questions 
until researchers have worked out a "feel" for the phenomena under discussion 
by means of extensive observation under field conditions. Ideally, as in 
other domains of theory/ practice , a dialectic evolves in which theoretical 
frameworks suggest directions for specific investigation, and the results of 
those investigations are fed back into clarifying the relevant constructs and 
adjusting the theoretical framework. 

Questions in this area tend to revolve around the philosophical concerns 
of what is true, what is good, and what is beautiful. 

I* Definition of instructional development . 

A. Meanings 

1. What meanings are attributed by users of the term (\nf erred from 
their usages)? 

2. What meanings can be inferred from observation of ID programs? 

3. What meanings are proposed authoritatively? 

B. Critical attributes 

1. As a hypothetical construct, what are the critical attributes of 
ID? 

2. To what extent does this hypothetical construct exist in reality? 
(Is anyone really doing ID?) 

II. The value of instructional development . 

A. Effectiveness 

1. Is ID more or less effective than alternative methods of planning 
instruction? 

2. Does the use of ID lead to superior learning? 

B. Cost/benefit 

1. What are the expected cost/benefit tradeoffs of ID? 

2. What are the most meaningful elements to count as "costs"? . . . 
as "benefits"? 

C. Side effects 

1. What are the short-term side effects of ID? 
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2* What are the long-term side effects of ID? • • . on learners? 
• • • on organizations? • • . on societies? 

Phenomena to Study and Methods of Observation 

Given the above agenda of questions to be answered, how would one go 
about attempting to cast light on these processes? What phenomena would the 
researcher wish to observe? Under what conditions of control? 

First, since ID is by definition a process involving interaction among 
humans and they necessarily operate within social systems, one of the most 
important classes of phenomena to be observed is the behavior or participants 
in the ID process. Participants include both those directly involved in ID 
projects and those within the organizational setting who have some relation- 
ship to ID, Gerlach (1984) argues that prescriptive rules can be developed 
by looking at the behavior of the developer as a dependent variable (p, 11)* 

Having determined that a major class of phenomena to be studied is that 
of the behavior of participants in ID, hov/ does one observe such behavior? 
There are at least three broad possibilities: 

I* Direct observation of behavior in vivo 

participant observation, nonpar ticipant observation, ethnography 

2* Indirect observation 

- perceptions of participants (e.g., through questionnaires), post 
hoc recollections of participants (e^g,, through interviews), 
examination of artifacts of ID activities (e,g,, ID documentation, 
organization records) 

3. Simulated observation 

contrived ID situations observed by means of experimental or 
quasi-experimental designs (e.g., comparison studies) • 

Another major class of phenomena to be studied is that of the effects of 
ID products on their intended audiences • ^That changes in knowledge, skill, 
and attitude result from exposure to developed instruction? From measurement 
of changes in ability or attitude we can infer the efficacy of different ID 
interventions • Questions of this sort lend themselves to the ttaditional 
research paradigms carried over from educational psychology, experimental, 
and quasi-experimental designs. 

The experimental paradigm applied to student achievement lends itself 
well to answering questions about the products of ID but less well to ques- 
tions about the processes of ID. For example, a review of research on forma- 
tive evaluation by Baker and Alkin (1973) reveals a rich vein of empirical 
research on this aspect of ID. They cite some two dozen studies in which 
various methods of formative evaluation have been tried out on real or simu- 
lated ID products. This element of the ID process — formative evaluation — 
perhaps because it lends itself to concrete testing wi**h actual learners, has 
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been one of Che mosc popular subjects for empirical study in the domain of 
ID. More recent studies (Kand aswamy et al « , 1976^ Nathenson, 1979 j Burton & 
Aversa, 1979; Golas, 1983; Wager, 1983; Israelite, 198A) continue to demon- 
strate the possibility of making inferences about optimal ID procedures based 
on the learning effects of those procedures. However, as we move backward in 
the ID process, away from the final product, the causal links become more and 
more tenuous. That is, it would be difficult to judge the efficacy, for in- 
stanc2, of a given task analysis technique based on the criterion of learner 
achievement because so many other intervening processes come between that 
task analys decision and the ultimate effect on the learner. For example, 
the task may have been analyzed beautifully, but the analysis may have been 
presented to the learner by means of a boring delivery /ehicle. Further 
methodological work is needed to clarify the current murkiness and lack of 
consensus regarding methods for evaluating these process phenomena • 

Methodological disputes in the domain of ID tend to revolve around trade- 
offs between internal and external validity. That is, employment of rigorous 
experimental designs with sophisticated statistical treatments enhances the 
internal validity (control over variables) at the expense of external validi- 
ty (applicability of findings Co real-life settings). The whole thrust of 
the naturalistic inquiry movement which is well under way in this field is to 
gain neaningf ulness , even at the sacrifice of generalizability, (The sacri- 
fice, of course, is not absolute; Schwen [1977] discusses ways in which 
generalizability can be enhanced, even in case studies,) 

The issue is not whether research in ID ought to adhere more to one para- 
digm than another. The issue is to select that paradigm and those methods 
that are most likely to cast light on the particular phenomena under scrutiny 
in a given study. It should be clear by now uhat even the rather restricted 
topl^ of instructional development encompasses phenomena ranging from learn- 
ers' cognitive processing of specific stimuli tiv the political ramifications 
of different project management schemes. The former may be studied profit- 
ably by experimental, psychological methodology, the latter by holistic, 
naturalistic observation. As Winn (1986) expressed aptly, 

I am bothered by the feeling I get that proponents of each class of me- 
thod are proposing alternatives to rather than complements to the other 
class of method. Given the eclectic nature of research questions in our 
discipline (and in Education generally), we must have at our disposal a 
whole battery of methods to deal with the different types of things we 
need to find out (p, 20), 

Thankfully, further steps down the path toward an agenda for research on ID 
can h -i guided by such aids as the criteria for inquiry advocated by Schwen 
(1977) and the logistical recommendations of Hannafin 91986), Researchers 
need not travel alone on this journey. 
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